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Background 

"I'm too much of a knucklehead to know this stuff" 

"Oh boy, confidence limits. I hate these. " 

"We use 90/50” 

"They don't use 90/50 they use 90/95.” 

“I should have used 90/50” 

"I defer my answer to the statistician” 

"I'm not a statistician. ” 

"90/50 POD means that there is a 50% chance that the true POD is greater than 
90% at that flaw size?" Responses: "No. ", and "Yes. " , rest of world gives blank 
stares. 

“Confusion over ‘common definitions’ continues to be an issue...” 



“We have been using 29 out of 29 clandestinely for years” 


Background (continued) 
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• The core issue here is that the NDE folks, nationwide, have delegated basic 
NDE POD statistical analysis to the statisticians. 

• This creates a divergence in the interrelationship between the physics of the 
inspection procedure and the POD statistics. 

• NDE community should not blindly accept statistical results due to lack of 
knowledge. 

• When NDE personnel defer explanations on statistical confidence bounds to 
others, it's like saying "I don't understand the error bars of my data". 

• This is not a good position. 

• We all need to learn and to speak the language of the other. 


This is the author's attempt to begin to bridge this gap. 


Using The Binomial Distribution 
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The binomial distribution describes the behavior of a count variable X if the following conditions 
apply: 

> The number of observations N is fixed. 

> Each observation is independent. 

> Each observation represents one of two outcomes ("success" or "failure"). 

> Use “green” or “red” to represent “Hit” or “Miss”, respectively. 

> The probability of ’’Hit" (POH) is the same for each outcome. 

If these conditions are met, then X has a binomial distribution. 


Set threshold 


Is it Green? 
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ANALOGY 

Various shades 
(amplitudes) may 
be due to crack 
size, signal 
amplitude, etc. 
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Hit/Miss 


By setting a threshold only two outcomes (Hit/Miss) observations are obtained. 


Using The Binomial Distribution (continued) 
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Number of Hits, X, after N trials 


O = Example 
values of POH 
as observed 
during a test. 


Use binomial distribution for now. 

Other distributions may be used if they can be demonstrated to be better. 


Probability of Hit (POH) Example 


• Start with 60 flaws in the group. 

• Each flaw has with the same probability of being observed as a Hit. 

• Make 60 observations. 

• If 58 Hits are observed, then the Probability of Hit is POH = 58/60 = 0.97 (the observed 
frequency) 

• This is an estimated POH since the true POH can only be approached by making an 
“infinite” amount of observations. 

• Now that the Probability of Hit is measured, what is the confidence in that value? 

• This is analogous to asking what are the error bars or uncertainty in measurements. 

• Confidence level is the measure of probability associated with a confidence interval 
expressing the probability of truth of a statement that the interval will include the 
parameter value. 




For NDE applications, the confidence bound of interest for Probability of Hit is the lower 
confidence bound. 


Confidence Level Example 


If the lower confidence limit (bound, value) = 0.90, then 

“There is a 95% chance that the true POH is greater than 90% for that flaw size” 


•*© ■* 



95% Chancstparameter value is \^ithin this region 
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Confidence Level Example (continued) 
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With X = 58 Hits after N = 60 trials, yielding 58/60 = 0.97 POH (the observed frequency), the 
lower confidence bound, p , may be obtained from* 



X 

X+(N~X+l)F a (f v f 2 ) 


FMJ. i)=l-83 


U=2(N-X+1)=14 

U=2X=108 


P, = 0-9 

a is, a priori, the confidence level, 95%, that we are requiring 
PJfvfi) ' s obtained from the F-distribution statistical table 


Note that the POH does NOT change if the confidence level is changed 


*lntroduction to Statistical Analysis”, W. J. Dixon and F. J. Massey Jr., 3 rd Edition, 1969, “Statistical Theory with Engineering Applications”, A. Hald, 

1 952, and “Recommended Practice for Demonstration of Nondestructive (NDE) Reliability on Aircraft Production Parts”, Ward D. Rummel, Materials 
Evaluation, vol 40, August 1982 


Confidence Level Example (continued) 
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P, = 0-9 


There is an 95% chance that the true POH is greater than 0.9 at that flaw size 

Or 

There is a 95% chance that the inspection system reliability is greater than 0.9 at that flaw size 

Or 

90/95 POD at that flaw size 


Directed Design of Experiments for Validating Probability of 
Detection of Inspection Systems (DOEPOD) 


DOEPOD Concepts 


> DOEPOD is a confidence driven approach that dynamically optimizes the 
class width to obtain the best lowest confidence value. 


> Select real initial flaw set or use simulated or existing inspection data. 
Guidelines are in the manual. (Smallest number of samples is 5). 


> Not all flaws are created equally; never identical; but they may be grouped by 
size, length, depth, etc. 


> Use moving average class width and class width size optimization to identify 
the best lower confidence bound. Class widths start at 0.001” and increase. 


Grouping of Flawed Specimens 
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• Flaws may be grouped into classes by size, length, depth, etc. The grouping may 
be any class width, e.g., 0.001”, 0.036”, 0.100”, etc. 


• If there are sufficient number of flaws, then a moving average may be used to 
dynamically group adjacent flaws into classes with widths (classwidths). 

> E.g., all flaws in the range 0.050” - 0.150” may be in a group, with the largest flaw being 
the identifier for the group. 

- Class width here is 0.100” 

- The next group may contain the range 0.049” - 0.149” if the moving average is 
from largest to smallest flaws. 


• POH needs to be determined for each flaw size grouping (number of flaws in each 
group is not necessarily the same) 


• Confidence bounds need to be determined for each flaw size grouping (both X and 
N vary for each group) 


What is the optimum class width? 


Effect of Grouping of 
Flawed Specimens 
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Observed Probability of Hit (POH) 



Important: When grouping 
flaws, the observed POH and 
confidence bound is assigned 
to the largest flaw in the group 



Size (inch), XI 


X Hit/Miss 

• POH 0.100" Classwidth 
a PL 0.100" Classwidth 


Analysis file name DOfcPOD VM.S G PC.xls 


DOEPOD 
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DOEPOD Concepts (continued) 


> Using POH lower confidence bound (value) as driver for directing DOE. 


> If 90/95 POD is reached at a then identify locations that need additional 
validation for larger flaw sizes. 


> If 90/95 POD is not reached then use best lower confidence value to identify 
where options are available to reach 90/95 POD. 


> Identify CASE of the data set. 


> Provide directions, depending on the CASE, on how to modify the DOE to 
continue to efficiently validate the inspection system. 


> Determine false call rate and associated confidence limit 


DOEPOD Parameters 
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Class length 

Inspection parameter (length, depth, etc.) 

Hit 

Flaw is detected 

Miss 

Flaw is not detected 

Need 

Add new samples to the existing set in order to reach the number of samples 
required at the class length 

LCL 

Lower confidence bound (value) of POH @ 95% confidence 

Opt. X P0H 

Optimum X P0H is identified for non-survey data sets. Optimum X P0H is the smallest class length 


Optimum X P0H and largest class width at which the minimum X POH = 1 occurs. Optimum X P0H may be more aggressive than 
optional, X PODopt , or X BestLCL , when the class width is constrained to the companion Optimum X POH class width listed. DOEPOD 


POH 

does not force use of Optimum X POH over X POD opt. or X Bes t lcl Stability has not been demonstrated at Optimum X POH therefore 
there is an additional risk that can not be satisfied. 

Probability of Hit (Number of Hits in Class Length/Total Number of Trials in Class Length) 

POD 

Probability of Detection (the true POD obtained if an infinite number of samples are used) 

Signal 

Amplitude 

Scalar amplitude output of NDE inspection system 

Survey Data 
Sets 

Survey Data Sets are data sets that have a sparse or disperse collection of samples. The moving 

average optimization has identified this set as having limited applications to moving average processing. An alternate 
optimization of X POH is used to provide guidance. 

Survey X POH 

Survey X POH is only identified for data sets determined to be Survey Data Sets. Survey X POH is the 
smallest class length and largest class width at which the minimum X P0H = 1 class length occurs. 

Survey X POH is the minimum class length at which X POD may be achieved when the class width is constrained to the companion 
survey class width listed. Survey X P0H is utilized in all cases in which it occurs. 
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DOEPOD Parameters (continued) 


X Bes tLCL Class length exhibiting the best LCL. The best class length is 

determined by increasing the moving average class width until a 
maximum LCL is obtained 

Xj Class length X at point “i” 

X L Largest class length in entire data set 

X m Class length near the mid-point between the largest and the smallest 

class lengths having no misses 

X POD Class length at which the lower confidence bound (value) is 0.90 or 
greater (90/95 POD) 

Xp 0H =i Class length where there are no misses above this class length 

Xpooopt Recommended optional existing smaller class length where X P0D may 
also be achieved if additional sample are added. 

X s Smallest class length in data set 

UCL Upper confidence value of the false call rate @ 95% confidence 


Case #1 (Best Case) 
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• 90/95 X P0D reached at a class length 

• Misses only below X P0D (and POH =1 everywhere greater than X P0D ) 

Directed Requirements for Validation of 90/95 X P0D 

^ Need samples at largest class length, X L 

> Need samples at mid-class length, X m = (X L - X P0D )/2 

> Option: Adding samples at X podopt may yield a new X P0D with a smaller class length. 

> If this is a survey data set, then only need to add samples at Survey X P0H (if listed) 

> Option: The user may add samples at Optimum X P0H The class width for all added samples at 
any class length is shown along with the Optimum X P0H 

> The range of validation may be expanded by adding samples at 2 X L , 4X L , 8X L , 16X L ,etc , if 
the current range of validation is too small. 


Case #1 (Best Case) 


Detection Probability 
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File Name - 
Data Set Name - 
Date &Time = 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 
Best LCL = 
Classwidth @ Best LCL - 
Classlength @ Best LCL = 
a(1 ) [Alpha] = 
a (2) [Beta] - 
Chi-Square - 


A8002L.Case1.XLS 
A8002L.Case1 (Eci-a-bf 

5/11/07 4:26 PM 
REACHED 


0.0040 

Inch 

0.0147 

Inch 

0.9120 

inch 

10.34 J 

1.94 

0.0023 

inch 


CASE 1 - 90/95 Xpod is VALIDATED from Xpod to XL 
w ben XL and Xm are satisfied. An alternate 90/95 Xpod is 
available ifXpodpot or Optimum Xpoh (if listed) is also 
satisfied. 


Su rvey /Optimum Xpoh - 0.0115-0.001 inch (need 

NTIAC 90% POD - 0.942 @ 0.010 

NTTAC 90; 95 POD = 0.952 @ 1015 


0.0000 


0.0500 


0.1000 


Note: Xpodopt is within one class width 
of Xpod. 


0.1500 0.2000 0.2500 

Size (inch), XI 

Analysis file name: DOE PQD.vl. 4.5.5.x Is 


0.3000 


0.3500 


0.4000 


False Call Rate = 

Largest Classlength 5 XL 
Samples Needed @ XL 
Classlength Mid-point s Xm 
Samples Needed @ Xm 
Smallest Classlength, Xs 
Samples Needed gXs 
New Smaller Classlength, Xss 
BestLCL Classlength, Xlcl 
Samples Needed eg Xlcl 
POH Classlength, Xpoh 
Samples Needed @ Xpoh 
New Largest Classlength , 2XL 
Xm is Near Verification Point 
Opt. POD classlength, Xpodopt 
Samples Needed gXpodopt 


with UCL @ 95% = 
0.342 
28 

0.161 
26 


0.0145 

29 


16 Samples) 

inch 

inch 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

Inch 


O Probability of Hit (POH) in Glass Range A Actual Lower Confidence Value (95%, F -distribution) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 






Case #2 
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• 90/95 X P0D reached at a class length 

• There are misses below X P0D and some misses above X P0D This is expected as X P0D nears 
the capability of the inspection system. 

• Since misses exist at class lengths X; above X P0D , then these greater lengths need to be 
validated, (i.e., The POH < 1 at class lengths Xj above X P0D point, X P0D , so these greater 
lengths need to be validated.) 


(a) Add samples of class length X; where POH<1 (TABLE A). Starting from largest 

class length, X; and work toward small class lengths until reaching an acceptable X P0D or 
reaching X P0D 

(b) Add samples of class length X, where POH=1 (TABLE B). Accept a larger X POD class length at 
any of the X, . This acceptance is valid as long as any existing larger class lengths where POH<1 
are shown [via (a) above] to be at 90/95 X P0D or greater. Acceptance of a larger X POD is not 
necessarily the X P0D capability for the inspection system. 

In summary, satisfy the smallest X P0D in Table B that is greater than the largest X POD in Table A, 
and/or the largest X POD in Table A. 


> If this is a survey data set, then only need samples at Survey X P0H (if listed), rather than at X POH=1 

> Option: If Optimum X POH < X P0H=1 then the user may add samples at Optimum X P0H rather than at X POH =i. 
The class width for all added samples at any class length is shown along with the Optimum X POH 


Directed Requirements 


> There are two (2) options that may be used to move this Case #2 toward Case #1 


> Need samples at largest class length, X L 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 


Case #2 


Detection Probability 



Size (inch), XI 

Analysis file name: DOE PGD.vl. 4.5.5.x Is 


File Name - 
Data Set Name - 
Date & Time = 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod = 
Actual Lower Confidence Value = 
Best LCL - 
Classwidth @ Best LCL - 
Classlength @ Best LCL - 
a(1) [Alpha] = 
a(2) [Beta] - 
Chi-Square - 


A8003L.Case2.XLS 

A8003L.Case2(Eci-a-bJl 

5/11/07 4:27 PM 

REACHED 
0.0040 inch 
0.0147 inch 
0.9120 



39.07 

8.21 

0.0035 


CASE 2 - 90/95 Xpod is reached at a class length. 

Further VALIDATION Is required. Recommend satisfying 
XL and the smallest Xpod In TABLE B that is greater than 
the largest Xpod in TABLE A, and/or the largest Xpod in 
Table A. 


Survey /Optimum Xpoh = 0.000 inch (need 

NT1AC 90% POD = 0.906 @ 0.010 

NTtAC 90/95 POD = 0.933 @ 3.015 

False Call Rate = with UCL @ 95% = 


Largest Classlength 5 XL - 
Samples Needed @ XL = 
Classlength Mid-point 5 Xm - 
Samples Needed @ Xm = 
Smallest Classlength, Xs - 
Samples Needed @ Xs = 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl - 
Samples Needed @ Xlcl - 
POH Classlength, Xpoh = 
Samples Needed @ Xpoh - 
New Largest Classlength , 2XL = 
Xm is Near Verification Points 
Opt. POD classlength, Xpodopt = 
Samples Needed @Xpodopt - 


0.342 

28 



Samples) 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Glass Range A Actual Lower Confidence Value (95%, F-distribution) 


Estimate of POH - Marquardt Gradient/Grid X 


Hit/Miss 




Number of Additional Samples Needed 


Case #2 

CASE 2 - 90/95 Xpod is reached at a class length. 

Further VALIDATION is required. 

Recommend satisfying XL and the smallest Xpod in TABLE B that is greater than the largest Xpod in TABLE A, and/or 
the largest Xpod in Table A. 


File Name = A8D03L.Case2.XLS 
Data Set Name = AB003L.Case2[Eci-a-bB] 

Directed DOE Options 


TABLE C 

Class Length (in) Additional Samples 



XL- 0.342 28 


Xm = 
Xs = 
Xss = 
Xlel = 
Xpoh - 
2X1- 
""Alternate Xm - 
Xpodopt - 


TABLE A* 

Selected class lengths 
with existing misses. 
Each point requires 
additional samples in or 
to achieve the Xpod 
listed. 


Xpod „ Class 

No. 

Length (in) 

Need 

0.1861 

44 

0.0437 

32 

0.0429 

26 

0.0421 

25 


TABLE B* 

Seiecfetf class 
lengths with no 
misses. Additional 
samples at these 
class lengths will 
achieve the Xpod 
listed. 


Xpod, Class 

No. 

Length (in) 

Need 

0.1992 

28 

0.0484 

26 

0.0464 

24 


Class Length (inch) 


O No Misses Observed ■ At Least One Miss OccuredAXL OXm OXs +Xss XXIcl XXpoh A2XL XXpod ♦Xpodopt 


Case #4 
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• 90/95 X P0D Not Reached 

• Best LCL is below 0.9 for the best class width group 

• No misses at or greater than class length exhibiting the best LCL, 
X Bes t i_ci_('- e -’ POH everywhere at or greater than class length 
currently having the best LCL, X BestLCL ) 


Directed Requirements 

y Need samples of X BestLCL in class length to achieve 90/95 X P0D at X BestLCL . X BestLCL 
may equal X L or X P0H=1 so that the number of samples listed at this class length are 
redundantly the same and only one set of samples is needed. 

> If this is a survey data set, then need to add samples at Survey X P0H (if listed), rather 
than at X P0H=1 

> Option: If Optimum X P0H < X P0H=1 then the user may add samples at Optimum X P0H 
rather than at X P0H=1 The class width for all added samples at any class length is shown 
along with the Optimum X P0H 


Case #4 


Detection Probability 
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File Name - 

C7002L.CASE4.XLS 

Data Set Name - 

C7002L.CASE4(Lpi-a) 

Date & Time = 

6/6/07 10:56 AM 

Xpod 90/95 Reached Anywhere? 

NOT REACHED 

Classwidth @ 90/95 Xpod - 

5:5:55:5 inGh 

Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 

■ inch 

Best LCL - 

0.8444 

Classwidth @ Best LCL = 

0.0970 inch 

Classlength @ Best LCL = 

0.2131 inch 

a(1) [Alpha] = 

24.24 

a(2) [Beta] = 

13.26 

Chi-Square - 

0.0007 


CASE 4 - 90/95 Xpod is not reached anywhere. 
Recommend satisfying XL and the greater of Xpoh or 
Xlcl. 


0.0500 


0.1000 


0.1500 0.2000 0.2500 

Size (inch), XI 

Analysis file name: DQEPGD.vl .4.5.6,xls 


0.3000 


0.3500 


0.4000 


Survey /Optimum >fcoh - 0.0976 -0.004 inch (need 

NT1AC 90% POD - 0.901 @ 0.350 

NTtAC 90/95 POD - 0.901 @ 1580 

False Call Rate = with UCL @ 95% - 

Largest Classlength 5 XL = 0.342 

Samples Needed @ XL - 27 

Classlength Mid-point , Xm = 

Samples Needed @ Xm = 

Smallest Classlength, Xs - 
Samples Needed @ Xs - 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 

Samples Needed @ Xlcl - 
POH Classlength, Xpoh = 

Samples Needed @ Xpoh - 
New Largest Classlength , 2XL - 
Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt = 

Samples Needed cgXpodopt - 


0.213 

11 

0.199 

13 


28 Samples) 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distribution) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 



• 90/95 X P0D Not Reached 


Case #5 


24 


• Best LCL is below 0.9 for the best class width group 

• There are misses at or greater than class length X BestLCL 

• There exists a class length, X P0H=1 , above which there are no 


• There are no misses for class lengths equal to greater than X L /3 
(i.e., Xp 0H =i < X L /3) 

• X P0H=1 < X l /3 so that POH is not fluctuating at larger class lengths. 
Use X P0H=1 as the trial X P0D 


> Need samples of X P0H=1 

> If this is a survey data set, then need to add samples at Survey X P0H (if listed), rather 
than at X P0H=1 

> Option: If Optimum X P0H < X P0H=1 then the user may add samples at Optimum X P0H 
rather than at X P0H=1 The class width for all added samples at any class length is shown 
along with the Optimum X P0H 

> Need 29 samples largest class length, X L 


misses. 


Directed Requirements 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 



Case #5 


25 


Detection Probability 
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File Name - 
Data Set Name - 
Date &Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod = 
Classlength @ 90/95 Xpod = 
Actual Lower Confidence Value - 
Best LCL - 
Classwidth @ Best LCL - 
Classlength @ Best LCL = 
a(1) [Alpha] - 
a(2) [Beta] = 
Chi-Square - 


C7001L.Case5.XLS 
C7001 L.CaseS(Lpi-a) 

5/11/07 4:31 PM 

NOT REACHED 
inch 


0.5493 

0.0040 

0.0738 

14.03 

4,74 

0.0086 


inch 

inch 

inch 


CASE 5 - 90/95 Xpod is nof reached anywhere. 
Recommend satisfying XL and Xpoh. 


Survey /Optimum Xpoh - 0.0881 -0.004 inch (need 28 Samples) 

NTIAC 90% POD - 0 @ 0.215 j nc h 

NTfAC 90/95 POD = 0.901 @ 1360 inch 


0.0500 


0.1000 


LCL 


> Optimum Xpoh Avail able; Using B&sf 


0.1500 0.2000 0.2500 

Size (inch), XI 

Analysis file name: DOEPOD.vl AS.S.xIs 


0.3000 


0.3500 


0.4000 


False Call Rate = with UCL (§ 

Largest Classlength , XL = 
Samples Needed @ XL - 
Classlength Mid-point , Xm - 
Samples Needed @ Xm = 
Smallest Classlength, Xs = 
Samples Needed @ Xs - 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 
Samples Needed @ Xlcl - 
POH Classlength, Xpoh - 
Samples Needed @ Xpoh = 
New Largest Classlength , 2XL = 
Xm is Near Verification Point- 
Opt. POD classlength, Xpodopt - 
Samples Needed @Xpodopt = 


95% - 
0.342 
28 


0.088 

28 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distribution) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Case #6 

• 90/95 X P0D Not Reached 

• Best LCL is below 0.9 for the best class width group 

• There are misses at or greater than class length X Best LCL 

• There exists a class length, X P0H =i, above which there are no misses. 

• There are are misses for class lengths greater than X L /3 (i.e., X P0H =i > X L /3) 

• XpoH=i - X L /3 so that POH may be fluctuating rapidly. 
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Directed Requirements 

> Need to expand current range of X L by adding new samples with class lengths of 2X L or greater 

> Need samples of X P0H=1 . X P0H=1 may equal X L so that the number of samples at this class length are 
redundantly the same and only one set of samples is needed. 

> Need 29 samples at largest class length, 2X L 

> If this is a survey data set, then need to add samples at Survey X P0H (if listed), rather than at X P0H=1 

> Option: If Optimum X P0H < X P0H=1 then the user may add samples at Optimum X P0H rather than at 
XpoH=i The class width for all added samples at any class length is shown along with the Optimum X P0H 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 


Case #6 


Detection Probability 
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File Name - 

D1003CD.Case6.XLS 

Data Set Name = 

D1003CD.Case6(CRAC 

Date & Time = 

5/11/07 4:39 PM 

Xpod 90/95 Reached Anywhere? 

NOT REACHED 

Classwidth @ 90/95 Xpod - 

inch 

Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 

■ inch 

Best LCL = 

0.6996 

Classwidth @ Best LCL - 

0.0130 inch 

Classlength @ Best LCL = 

0.0360 inch 

a(1 ) [Alpha] = 

9.04 

a (2) [Beta] = 

1.99 

Chi-Square - 

0.0029 


0.0000 0.0200 0.0400 


0.0600 0.0800 0.1000 0.1200 0.1400 0.1600 0.1800 0.2000 

Size (inch), XI 

Analysis file name: DOEPQD.vl AS.S.xIs 


CASE 6- 90/95 Xpod is not reached anywhere. 
Recommend satisfying XL, Xpoh, and 2XL. 


Survey/ Optimum Xpoh - 0.1260 -0.006 inch (need 

NT1AC 90% POD - 0*911 © 0.035 

NTIAC '90/95 POD = 0.906 © 0.050 

False Cal! Rate = with UCL @ 95% - 

Largest Classlength , XL = 0.178 

Samples Needed @ XL = 26 

Classlength Mid-point , Xm = 

Samples Needed @ Xm - 
Smallest Classlength, Xs - 
Samples Needed @ Xs - 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl - 
Samples Needed @ Xlcl = 

POH Classlength, Xpoh - 
Samples Needed @ Xpoh = 

New Largest Classlength , 2XL - 
Xm is Near Verification Point .= 

Opt. POD classlength, Xpodopt = 

Samples Needed @Xpodopt - 


0.176 

26 

0.356 


26 Samples) 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distributi on) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Case #7 


28 


• 90/95 X P0D Not Reached 

• Best LCL is below 0.9 for the best class width group 

• There are misses at or greater than class length X Best LCL 

• There does NOT exist a class length, X P0H=1 , above which there are no 
misses. 

• POH may be fluctuating rapidly 

• There may be no hits anywhere 

Directed Requirements 


> Inspection system may not be appropriate for meeting inspection criteria 

> If this is a survey data set, then need to add samples at Survey X P0H (if listed) 

> Option: The user may add samples at Optimum X P0H The class width for all added samples at any 
class length is shown along with the Optimum X P0H 

> Need to expand current range of X L by adding new samples with class lengths of 2X L or greater 

> Need 29 samples at largest class length, 2X L 


Case #7 


Detection Probability 



Pile Name - 

AB003(3)L. Case7.xls 

Data Set Name - 

AB003(3)L.Case7(CK. 1 

Date &7ime - 

5/11/07 4:29 PM 

Xpod 90/95 Reached Anywhere? 

NOT REACHED 

Classwidth @ 90/95 Xpod - 

nCh 

Classlength @ 90/95 Xpod = 
Actual Lower Confidence Value - 

M inch 

Best LCL = 

0.7206 

Classwidth @ Best LCL - 

0.0060 inch 

Classlength @ Best LCL - 

0.1000 inch 

a(1) [Alpha] = 

1.07 

a(2) [Beta] = 

0.97 

Chi-Square - 

0.1997 


CASE 7 - 90/95 Xpod is not reached anywhere. 
Recommend satisfying 2XL and Optimum Xpoh (if 
ilisted). 


Size (inch), XI 

Analysis file name: D0EPGD,vl4.5 t 5.xl5 


Survey /Optimum X>oh - 0.000 inch (need 

NTIAC '90% POD = 6 

NTTAC 90/95 POD = ® 

False Call Rate = with UCL @ 95% = 

Largest Classlength , XL - 
Samples Needed @ XL = 
Classlength Mid-point , Xm = 

Samples Needed @ Xm = 

Smallest Classlength, Xs - 
Samples Needed @ Xs = 

New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl - 
Samples Needed @ Xlcl - 
POH Classlength, Xpoh = 

Samples Needed @ Xpoh = 

New Largest Classlength , 2XL - 
Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt - 
Samples Needed @Xpodopt - 


2.376 



Samples) 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distributi on) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Survey and Optimized X P0H Data Sets 
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• This data set has insufficient number of samples for unconstrained moving average 
optimization 

• The moving average optimization has determined that there is a moving average 
class width for which the smallest X P0H =i class length is identified. The Survey and 
Optimum X P0H class lengths and class widths are identified on the charts as 
Survey/Optimum X P0H - 

> For example, the listing: 

Survey/Optimum X P0H = 0.0500 - 0.015 inch (need 18 samples) 

indicates that a class width of 0.015” has been used and the Survey or 
Optimum X P0H occurs at 0.0500”, and that 18 additional samples may be added 
to achieve X P0D add at that class length. The added samples should have sizes 
that range anywhere between 0.0500” and 0.035”, inclusively. 

• If Survey/Optimum X P0H is pursued, then the class width for all added samples at 
any other class length is to be the same as that for Survey/Optimum X P0H . 

Directed Requirements 

> Need samples at Survey/Optimum X P0H 

> Need samples at X L 

> Survey results are not available when user sets the class width 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 


Case #5 - Survey 


Detection Probability 
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File Name - 
Data Set Name - 
Date & Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength g 90/95 Xpod - 
Actual Lower Confidence Value - 
Best LCL = 
Classwidth g Best LCL = 
Classlength eg Best LCL = 
a(1) [Alpha] = 
a(2) [Beta] - 
Chi-Square = 


Survey.Case5.xls 
Survey.Case5(Survey i 

5/14/07 9:23 AM 
NOT REACHED 



inch 

inch 

02236 

0.0390 

inch 

0.4000 

inch 

5.36 


4.65 


0.0296 



CASE 5 - This is a survey data set 90/95 Xpod is not 
reached anywhere. Recommend satisfying XL and 
Survey Xpoh (i f li ste d) 


Survey /Optimum Xpoh - 0.4600 -0.039 inch (need 

NT1AC 90% POD s @ 

NTIAC 90/95 POD - ® 


False Call Rate - 


with UCL @ 95% - 
3.000 
26 


-> Survey Data Set; System Set Class 
Width 


0.5000 1.0000 1.5000 2.0000 

Size (inch), XI 

Analysis file name: DOEPOD.vl. 4.5j6.xls 


2.5000 


3.0000 


3.5000 


Largest Classlength s XL - 
Samples Needed g XL = 
Classlength Mid point , Xm = 
Samples Needed g Xm = 
Smallest Classlength, Xs - 
Samples Needed g Xs - 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 
Samples Needed g Xlcl = 
POH Classlength, Xpoh = 
Samples Needed g Xpoh - 
New Largest Classlength , 2XL = 
Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt - 
Samples Needed gXpodopt - 


0.460 

26 


26 Samples) 
inch 
inch 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F -distribution) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 
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Does the fastener 


DOEPOD Example 

type affect the capability of the inspection system? 


Detection Probability 



Size (inch), XI 

An aiy s i s fi i e nam e: DO E P 0 D. vl A. 5, 6. x Is 


File Name = 
Data Set Name - 
Date & Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod = 
Actual Lower Confidence Value - 
Best LCL = 
Classwidth @ Best LCL - 
Classlength @ Best LCL - 
a(1) [Alpha] = 
a(2) [Beta] = 
Chi-Square = 


6/5/07 8:10 AM 

REACHED 
0,0380 inch 

0,1000 inch 
0:0070 



23.11 

6.34 

0.0000 


inch 

inch 


CASE 1 - 90/95 Xpod is VALIDATED from Xpod to XL 
wften XL and Xm are satisfied. An alternate 90/95 Xpod is 
available If Xpodpot or Optimum Xpoh (if listed) is also 
satisfied. 


Survey/Optimum Xpoh = 0.6620 -0.038 inch (need 22 Samples) 


NTlAC 00% POD = 
NTIAC 00/05 POD - 
False Call Rate = 


© 

with UCL @ 05% = 


Largest Classlength , XL = 
Samples Needed @ XL - 
Classlength Mid-point 5 Xm - 
Samples Needed @ Xm = 
Smallest Classlength, Xs - 
Samples Needed @ Xs = 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 
Samples Needed @ Xlcl - 
POH Classlength, Xpoh = 
Samples Needed @ Xpoh - 
New Largest Classlength , 2XL - 
Xm is Near Verification Point- 
Opt. POD classlength, Xpodopt - 
Samples Needed @Xpodopt - 


0.200 

15 

0.125 

5 


0.0620 

22 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distribution) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 



DOEPOD Example (continued) 
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Detection Probability 



File Name = 
Data Set Name = 
Date & Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod = 
Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 
Best LCL = 
Classwidth @ Best LCL - 
Classlength @ Best LCL = 
a(1) [Alpha] = 
a (2) [Beta] = 
Chi-Square = 


6/5/07 8:22 AM 

NOT REACHED 
inch 


0.6146 

0.0300 

0.1600 

2.41 

0.05 

0.01 57 


inch 


inch 

inch 


CASE 7 - 90/95 Xpod is not reached anywhere. 
Recommend satisfying 2XL and Optimum Xpoh (if 
ilisted). 


Survey/Optimum Xpoh = 0.000 inch (need 

NTtAC 90% POD - @ 

NTTAC 90/95 POD s ® 

False Call Rate = with U CL @ 95% = 

Largest Classlength s XL - 
Samples Needed @ XL = 
Classlength Mid-point , Xm - 
Samples Needed @ Xm - 
Smallest Classlength, Xs = 

Samples Needed @ Xs = 

New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 

Samples Needed @ Xlcl = 

POH Classlength, Xpoh = 

Samples Needed @ Xpoh - 
New Largest Classlength , 2XL = 0 400 

Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt = 

Samples Needed @Xpodopt - 


Samples) 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 
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DOEPOD Example (continued) 


Does the fastener type affect the capability of the inspection system? 


Optimum Xpoh (Best 
Xpod that could be 
achieved if test 


Multi- 

Fasteners Inspector 

90/95 Xpod (in) 

specimens are added) 
(in) 

DOEPOD 
vl .4.5.6 

5-Jun-07 

1 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

2 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

3 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

4 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

5 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

6 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

7 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

8 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

9 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

10 

0.150 

0.090 

CASE 1 

90/95 Xpod reached 

11 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

12 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

13 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

14 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

15 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

16 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

17 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

18 

0.200 

0.125 

CASE 1 

90/95 Xpod reached 

19 

0.200 

0.125 

CASE 1 

90/95 Xpod reached 





90/95 Xpod reached but 

20 

0.100 

see table A 

CASE 2 

miss at largest flaw size 

21 

not reached 


CASE 7 

Miss at largest flaw size 

22 

not reached 


CASE 7 

Miss at largest flaw size 



Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 
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DOEPOD Example (continued) 
All Inspectors 


Detection Probability 



Size (inch), XI 

Analysis file name: DOEPOD. vl A 5. 6.x Is 


File Name = 
Data Set Name - 
Date & Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 
Best LCL = 
Classwidth @ Best LCL - 
Classlength @ Best LCL - 
a(1) [Alpha] = 
a(2) [Beta] = 
Chi-Square = 


5/31/07 5:15 PM 

REACHED 
0.0010 inch 

0.1000 inch 
0.9260 



8.33 

2.33 
0.0004 


inch 

inch 


CASE 1 - 90/95 Xpod is NOW VALIDATED from Xpod to 
XL. An alternate 90/95 Xpod Is available if Xpodpot or 
Optimum Xpoh (if listed) is also satisfied. 


Survey /Optimum Xpoh = 0.000 inch (need Samples) 


NTlAC 90% POD ^ m 

NTIAC 90/95 POD = © 

False Call Rate = with UCL @ 95% = 

Largest Classlength , XL - 0.200 

Samples Needed @ XL - o 
Classlength Mid-point , Xm = 0 . 15 O 

Samples Needed @ Xm = 

Smallest Classlength, Xs - 
Samples Needed @ Xs = 

New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 

Samples Needed @ Xlcl = 

POH Classlength, Xpoh = 

Samples Needed @ Xpoh = 

New Largest Classlength , 2XL = 

Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt - 0.0950 

Samples Needed @Xpodopt - 29 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

Inch 

inch 


O Probability of Hit (POH) iin Class Range A Actual Lower Confidence Value (95%, F-distribution) 


Esti mate of POH - Marquardt Gradi ent/Gri d X 


Hit/Miss 
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False Calls 


• False Calls are handled similarly except the upper confidence limit* 


FalseCallRate= 


Number of False Calls (X) 
Number of False Call Opportunites (N) 


p (X + l )F a (f l ,f 2 ) |/i =2(X + Y) 1 

“ (N-X) + (X + \)F a (f v f 2 ) ’ \f 2 =2(N — X)j 


\ is used 
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False Calls 


• Test samples with no flaws present may be included in DOEPOD for determination 
of false call rate and the upper confidence value of the false call rate at 95% 
confidence. For test samples with no flaw present, enter flaw size of 0.00001” 

• False call rate may be explored and optimized by adjusting signal amplitude 
threshold. 

• Warning: May reach 90/95 X P0D at cost of increasing false call rate. Need to know 
what false call rate is acceptable. 

• False calls rate should not be accepted as is without first addressing the cause of the 
false call and identifying procedures to remove false calls. May need to modify or add 
inspection protocols. 


DOEPOD Data Entry 
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Hit / Miss Data 


ID Number 

CRACK SIZE (inches} 

DEPTH 

HIT/MlSS (0 or 1 

Signal Amplitude 
Measured (Arbitrary 
Units} 

SIGNAL 

TREASHOLD 


0.342 


100 




0.251 


100 




0.242 


100 




0.213 


100 




0.200 


100 




0.199 


100 




0.106 


100 




0.104 


0 




0.169 


100 




0.166 


0 




Signal Amplitude Data 


ID Number CRACK SIZE (inches} 

DEPTH HIT/MISS (0 or 100} 

Signal Amplitude 
Measured (Arbitrary 
Units} 

SIGNAL 

TREASHOLD 

0.342 


0.5 

0.25 

0.251 


0.4 


0.242 


0.3 


0.213 


0.2 


0.206 


0.1 


0.199 


0.05 


0.106 


0.025 


0.104 


0.01 


0.169 


0.025 


0.166 


0.01 



DOEPOD REQUIREMENTS 
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• Inspection processes are to be under control and fixed. 

• Multiple inspection processes may be used on the same set of test samples with the constraint that 
Directed DOE POD is to be executed for each process separately. When multiple inspection processes 
or systems are used, the resulting directed sample requirements may be overlapping. In this situation, 
the user is to keep the non-overlapping directed sample requirements applied to the appropriate 
inspection process, while utilizing overlapping directed sample requirements for the multiple processes in 
order to minimize the number of generated test samples. 

• There are to be an equal number of unflawed samples during any test. 

• There are to be more than two (2) samples at different class lengths. 

• A moving average that groups flaws of similar size is used to optimize the lower confidence value. This 
moving average and best lower confidence bound (value) optimization will be invoked if there are more 
than four (4) samples at different class lengths. 

• Flaw sizes must be greater than 0.00001” 

• Test samples with no flaws present may be included for determination of false call rate and the upper 
confidence value of the false call rate at 95% confidence. For test samples with no flaw present, enter 
flaw size of 0.00001” 

• The maximum number of test samples is 1 999. 

• Be prepared to generate, inspect, and evaluate test samples during the NDE technology capability 
determination. 

• Validated 90/95 X P0D is obtained when the user has reached and satisfied the sample requirements of 
Case 1 . That is, there is a 95% chance that the probability of detection of the system is greater 
than 90% for class lengths in the range 90/95 X POD to XL. 
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SUMMARY 


• Concept for Binomialization of Test Data 

• Process for determining observed probability of hit (POH) and associated confidence limits 

• Utilization of moving average to group flaws and flaw class width optimization 

• Identification of POD CASES and directed actions needed to validate inspection systems. 

• False call rate and confidence 

• DOEPOD Data Entry 

• DOEPOD Beta (2,500 lines of code, PC and MAC, limited distribution) 

• Future work: DOEPOD upgrades 

> Interface with predicted POD MIL-HDBK-1823 

- companion tool 

> Address very limited data sets when 90/95 X P0D 
can never be reached, and communicating those risks. 
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• DOEPOD Probability of Hit estimating curve is for visualization only 
and not used in the DOEPOD analysis. 

• Probability of Hit estimating curve is not to be used for validation or for 
justification of validation. 

• The default function used in DOEPOD is: 

POH = Exp(a(1) + a(2) * Log(xl)) / (1 + Exp(a(1) + a(2) * Log(xl))) 

a(1) = a 
a(2) = 

xl = Flaw size 


• Other multi-parameter functions may be used 


